fix(podcasts): restore the language instruction and stop the prompts handing the model a copyable skeleton - #1334
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…handing the model a copyable skeleton (lfnovo#1238) Two independent defects in prompts/podcast/*.jinja, plus the failure hints that made both of them hard to diagnose. Split out of lfnovo#1239 at review request; the voice pre-flight from that PR stays there. Language -------- These templates shadow podcast-creator's bundled ones - the library resolves Path.cwd()/prompts/podcast/<name>.jinja before its own package resources, and this app configures only profiles - and the app's copies never referenced `{{ language }}`. The value is plumbed correctly end to end, so an episode profile set to he-IL produced an English outline and English segment titles: EpisodeProfile.language looked supported and did nothing. The bundled block is restored in both templates, and a regression test fails when a variable the bundled template uses is missing from the app's copy - the drift that lost it in the first place. Copyable skeleton ----------------- Both templates showed a fill-in JSON skeleton (`"speaker": "[Actual Speaker Name]"`, a bare `...`) inside a ```json fence while also instructing the model not to use fences. Models returned the skeleton verbatim, which failed podcast-creator's speaker-name validation and aborted the episode - discarding the outline and every segment already generated. The example now carries the episode's real speaker names (serialized with tojson, so a name containing a quote can't break it) and written-out sample values, so a verbatim copy is valid output. Placeholders and truncation are banned explicitly, the contradictory fence and the second copyable example are gone, and the example is labelled a two-entry excerpt that restates the required turns/num_segments count so it can't read as a finished answer. No sample is rendered at all when a language is set: any hard-coded sample is English, and a copied English line in a Hebrew episode goes straight to TTS. The structure is stated in prose there, and format_instructions still carries the schema. Hints ----- explain_generation_failure() maps a failure to the hint that fits it. The GPT-5 extended-thinking note was the only one, keyed on `Invalid json output` / `Expecting value`, so `Invalid speaker name` got nothing and a truncated Gemini response was advised to switch to gpt-4o. Placeholder speaker names, unusable voices and truncated output now each get their own explanation.
…nd one (lfnovo#1238) Review follow-ups on this PR. - The hint mapper never ran for the failures it was written for. `except ValueError: raise` sits ahead of the handler that attaches hints, and both LangChain's OutputParserException and json.JSONDecodeError are ValueError subclasses - so a placeholder speaker name, `Invalid json output` and a truncated response all left untouched. That includes the pre-existing GPT-5 extended-thinking note, which has been dead for exactly the case it describes. The ValueError branch now attaches the hint and re-raises a ValueError, keeping the command layer's permanent-failure contract intact. - `Requested entity was not found` no longer claims the voice is at fault. Google returns it for any missing resource, including a model id on the outline or transcript call, so the hint now names both candidates and says which is likelier when audio had already started. The voice-specific hint stays keyed on `Voice name ... is not supported`, which does name the voice. - A solo speaker profile renders a one-entry example; the prose called it a two-entry excerpt and told the model to return more than "the two shown". Both now follow speakers|length. - The template drift check counted Jinja's `not` as a variable, so `{% if not language %}` would fail the check the moment one template negated a test the other didn't.
6fb1510 to
9228bb9
Compare
|
All four cubic findings were valid; fixed in The first one is worse than reported, and it's the important one.
Solo-speaker contradiction — fixed; the excerpt wording follows
772 tests pass, ruff and mypy clean; the two new hint assertions fail against |
2 participants
Description
First of the two PRs #1239 was split into, per @lfnovo's review. This one is the low-risk half: the prompt templates and the failure hints. The voice pre-flight stays in #1239 for its own review.
1.
EpisodeProfile.languageis inert.prompts/podcast/{outline,transcript}.jinjashadow podcast-creator's bundled templates — the library resolvesPath.cwd()/prompts/podcast/<name>.jinjabefore its own package resources, and this app only callsconfigure("speakers_config"/"episode_config")— and the app's copies never referenced{{ language }}. The value is plumbed correctly end to end (graph.py:100resolves it,graph.py:133maps the code to a name,nodes.pypasses it), so a profile set tohe-ILproduced an English outline and English segment titles. The field looked supported and did nothing.Restored in both templates. A regression test now fails when a variable the bundled template uses is missing from the app's copy — the drift that lost it.
2. The prompts handed the model a skeleton it could copy. Both templates showed a fill-in JSON skeleton (
"speaker": "[Actual Speaker Name]"plus a bare...) inside a```jsonfence, while also instructing the model not to use fences. Gemini returned the skeleton verbatim, which failed podcast-creator's speaker-name validation and aborted the episode at segment 2/6 — discarding the outline and the segment already generated:The example now carries the episode's real speaker names (via
tojson, so a name containing a quote can't break it) and written-out sample values, so a verbatim copy is valid output. Placeholders and truncation are banned explicitly, the contradictory fence and the second copyable example are gone, and the example is labelled a two-entry excerpt restating the requiredturns/num_segmentscount so it can't read as a finished answer.Per your last note, no sample is rendered when a
languageis set — any hard-coded sample is English, and a copied English line in a Hebrew episode goes straight to TTS. The structure is stated in prose there, andformat_instructionsstill carries the schema.3. Failure hints.
explain_generation_failure()maps a failure to the hint that fits. The GPT-5 extended-thinking note was the only one, keyed onInvalid json output/Expecting value, soInvalid speaker namegot nothing and a truncated Gemini response was told to switch togpt-4o.Related Issue
Part of #1238. Split from #1239.
Type of Change
How Has This Been Tested?
uv run pytest)Test Details: 768 tests pass,
ruff check .andmypyclean. New:tests/test_podcast_prompt_templates.py(renders both templates: language block present/absent, example parsed withjson.loadsfor speaker fidelity and escaping, no placeholder skeletons, no sample at all when a language is set, and the bundled-vs-app variable drift check) andtests/test_podcast_error_hints.py.Verified the tests are genuine regressions against pre-fix templates:
[Actual Speaker Name]present,{"segments": [...]}present, no language block withlanguage="Hebrew", andlanguagereported missing from both templates.Not smoke-tested end to end from this branch — the end-to-end Hebrew run (6 segments, 42 lines, MP3, 474 s) was on the reporting deployment with these templates bind-mounted.
Design Alignment
Explanation: The pipeline silently assumed English output; a non-English deployment got a podcast in the wrong language with no signal. No new configuration or endpoints — the prompts just stop contradicting themselves.
Checklist
Code Quality
Testing
ruff check . --fixuv run python -m mypy .Documentation
/docs—docs/7-DEVELOPMENT/podcasts.mdgains a "Prompt templates shadow podcast-creator's" section: the resolution order, which variables each template receives, and why no sample is rendered for a non-English runDatabase Changes
Breaking Changes
Additional Context
The deeper question from #1238 stands and is deliberately not answered here: these templates have diverged from the library's in both directions — they gained the solo-speaker branches and the
<think>-tag rules and lost the language block. Either the app-only features get upstreamed into podcast-creator andprompts/podcast/is deleted, or the fork is documented as intentional. The drift test makes the next divergence fail loudly in the meantime.Pre-Submission Verification